Projection of Images onto Tangible User Interfaces

ABSTRACT

A surface computing device is described which has a surface which can be switched between transparent and diffuse states. When the surface is in its diffuse state, an image can be projected onto the surface and when the surface is in its transparent state, an image can be projected through the surface and onto an object. In an embodiment, the image projected onto the object is redirected onto a different face of the object, so as to provide an additional display surface or to augment the appearance of the object. In another embodiment, the image may be redirected onto another object.

BACKGROUND

Surface computing devices have been developed which comprise a surfacewhich is used both for displaying the graphical user interface and foruser input. The surface computing devices detect the user's fingers onthe surface or detect real, tangible objects which are manipulated by auser and this is referred to as a ‘tangible user interface’ (TUI). In anexample, the objects may be gaming pieces which may be moved by a userand the motion can be detected by the surface computing device. Thesurface computing devices may be designed for use by a single user orthey may be multi-user devices.

There are several techniques which have been developed for tracking ordetecting objects on the surface, for example, using cameras to imageobjects from above the surface (a ‘top-down’ configuration) or usinglight sources to illuminate the surface from below and cameras to detectlight reflected by objects in contact with the surface (a ‘bottom-up’configuration). Another technique relies on frustrated total internalreflection (FTIR) to cause scattering of light when a fingertip is incontact with the surface, and this scattered light is detected by acamera below the surface.

The embodiments described below are not limited to implementations whichsolve any or all of the disadvantages of known surface computingdevices.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding to the reader. This summary is not anextensive overview of the disclosure and it does not identifykey/critical elements of the invention or delineate the scope of theinvention. Its sole purpose is to present some concepts disclosed hereinin a simplified form as a prelude to the more detailed description thatis presented later.

A surface computing device is described which has a surface which can beswitched between transparent and diffuse states. When the surface is inits diffuse state, an image can be projected onto the surface and whenthe surface is in its transparent state, an image can be projectedthrough the surface and onto an object. In an embodiment, the imageprojected onto the object is redirected onto a different face of theobject, so as to provide an additional display surface or to augment theappearance of the object. In another embodiment, the image may beredirected onto another object.

Many of the attendant features will be more readily appreciated as thesame becomes better understood by reference to the following detaileddescription considered in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

The present description will be better understood from the followingdetailed description read in light of the accompanying drawings,wherein:

FIG. 1 is a schematic diagram of a surface computing device;

FIG. 2 shows a flow diagram of an example method of operation of asurface computing device;

FIGS. 3 and 4 show schematic diagrams of various passive objects whichmay be used with a surface computing device;

FIG. 5 shows a schematic diagram of an active object and a surfacecomputing device;

FIGS. 6-8 show schematic diagrams of surface computing systems which maybe used for detecting user input;

FIG. 9 is a schematic diagram of another surface computing device;

FIG. 10 shows schematic diagrams of optical arrangements for correctingdistortion;

FIG. 11 shows a flow diagram of another example method of operation of asurface computing device; and

FIG. 12 illustrates an exemplary computing-based device in whichembodiments of the methods described herein may be implemented.

Like reference numerals are used to designate like parts in theaccompanying drawings.

DETAILED DESCRIPTION

The detailed description provided below in connection with the appendeddrawings is intended as a description of the present examples and is notintended to represent the only forms in which the present example may beconstructed or utilized. The description sets forth the functions of theexample and the sequence of steps for constructing and operating theexample. However, the same or equivalent functions and sequences may beaccomplished by different examples.

FIG. 1 is a schematic diagram of a surface computing device 100 whichcomprises a projector 101 which is located behind a switchable surface102, i.e. the projector 101 is on the opposite side of the switchablesurface to a user (not shown in FIG. 1). The switchable surface 102 ishas two modes of operation: a ‘display mode’, in which the surface issubstantially diffuse (to visible light) and any rear-projected image orother graphical data is displayed on the surface, and a ‘projectionmode’, in which the surface is substantially transparent (to visiblelight) and any rear-projected image (or other graphical data) isprojected through the surface. If an object, such as object 103, isplaced on (or near) the surface 102, the image may be projected into thebottom face 104 of the object when the device is in projection mode.

For the purposes of explanation only, the description herein refers tothe projection of graphical data through and/or on to the surface (e.g.by projector 101). This graphical data may comprise any form of image ordata, including digital or analog data. It will be appreciated that anyform of graphical data may be projected by the projector through thesurface and the choice may be application dependent. In some examples,text or a graphical user interface (GUI) may be projected (which maycomprise text and/or images), in other examples, an image, which may bestill or moving, may be projected, and in further examples a singlecolor or other light pattern (e.g. a structured light pattern) may beprojected.

FIG. 2 shows a flow diagram of an example method of operation of thesurface computing device 100. In this example, the switchable surface102 is initially in display mode and a graphical user interface for thesurface computing device or any other graphical data is projected ontothe surface (block 201). The surface is then switched into projectionmode (block 202), i.e. it is switched from substantially diffuse tosubstantially transparent to visible light, and graphical data isprojected through the surface onto an object (block 203). The switchablesurface may then be switched back to display mode (block 204) and themethod may be repeated.

The switching of the surface (in blocks 202 and 204) may be done at anyrate. In an example, the surface may be switched at a rate which exceedsthe threshold for flicker perception (e.g. at 60 Hz). At such a rate auser will see both the GUI (or other graphical data) on the surface andthe graphical data (which may be another GUI) projected onto the object.Such an apparatus and method therefore enables display of two differentelements of graphical data, GUIs or other forms of data substantiallysimultaneously. These two elements may be totally unrelated and may beindependently controlled. In other examples the same graphical data maybe projected on to and through the surface (e.g. to project onto asecond display surface in addition to the switchable surface).

In order to display different graphical data onto the surface andthrough the surface, a projector may be used which can switch at a highenough rate between elements of graphical data (e.g. between images) andthe projector may be synchronized with the switching of the surface(e.g. in blocks 202 and 204). Alternatively, the system may comprise asecond projector 106 and a switchable shutter (or filter) 107, 108 maybe provided in front of each projector 101, 106. In such an embodiment,the shutters 107, 108 may be switched in synchronization with theswitchable surface 102 and the projectors 101, 106 may projectcontinuously (or be switched at a lower rate). Alternatively, aprojector 101 may be provided to project graphical data through thesurface whilst an alternative display means, such as an LCD (liquidcrystal display) panel may be used in display mode to project graphicaldata onto the surface in display mode. In such an example, the projector101 may act as the backlight for the LCD panel or a separate backlightmay be provided. For the purposes of the following explanation only,projectors will be used to provide the graphical data in the displaymode, although it will be appreciated that in other examples, anydisplay means which can be projected through may be used.

The dual projection capabilities of the surface computing device 100 canbe used to create interesting layering and magic lens effects. Insteadof projecting two entirely unrelated images, the graphical dataprojected through the surface may be visually connected to the graphicaldata being projected on the surface. For example, the graphical dataprojected on the surface could be an image of a car, with graphical datacomprising an associated image that reveals the inner workings of thecar being projected through the surface. In this example scenario, if auser passes a piece of translucent material (or other object) over thecar, it will reveal this otherwise hidden information, creating a twolayered effect. Different translucent objects of varying forms andshapes can be used to exploit this capability, each effectively actingas a physical magic lens. The object does not have to be resting on oreven in contact with the surface. It can be lifted off the surface andthe through surface projected graphical data is maintained. The objectmay be manipulated in six degrees of freedom. Further examples aredescribed below.

The projectors 101, 106 may be any kind of projector and examplesinclude, but are not limited to, LCD, LCOS (liquid crystal on silicon),DLP (Digital Light Processing™) and laser projectors. The projectors maybe fixed or steerable. The switchable shutters 107, 108 (orfilters/mirrors) may be any kind of switchable shutter and an example isa ferroelectric LCD shutter.

The switchable surface 102 may comprise a sheet of Polymer StabilizedCholesteric Textured (PSCT) liquid crystal or Polymer Dispersed LiquidCrystal (PDLC). These materials can be electrically switched betweensubstantially diffuse and transparent states by applying a voltage. PSCTis typically capable of being switched at higher rates than PDLC andPSCT can be switched at rates which exceed the threshold for flickerperception (e.g. it may be switched at around 60 Hz).

In an example implementation using PSCT, the surface may be switched at60 Hz, with each cycle consisting of around 8.3 ms when 150V is appliedto the screen to make it clear followed by 8.3 ms with no appliedvoltage, at which point it returns to its natural diffuse state. Theexact proportion of time in each state (i.e. the duty cycle) can bevaried according to specific needs of the system design. Increasing thediffuse interval at the expense of the clear interval, for example, willincrease display brightness on the surface at the cost of reducingbrightness of the through projection. It will also decrease theavailable light to the camera for imaging through the surface. In thisexample, the polarity of the 150V may be reversed on alternate cyclesand a driver circuit based on an H-bridge architecture may be used, witheach side of the switchable surface connected to one half-bridge,capable of switching between 0 and +150V. A potential of 0V, +150V or−150V may therefore be applied across the PSCT-LC depending on whetherneither, the left or the right half-bridges are enabled (respectively).Each half-bridge may be implemented as a complementary emitter follower,made from NPN and PNP power audio transistors. These transistors arecapable of delivering the high current (˜4 A) required to switch thesurface, which is effectively a nonlinear capacitor of around 6 μF,quickly enough. The power output stage may be driven through anadditional current gain stage. Electrical isolation between thehigh-voltage circuitry and the rest of the system may be achievedthrough the use of an opto-coupled level shifter. It will be appreciatedthat this is just one possible implementation and is described by way ofexample only.

Alternatively, the switchable surface may use any other technology orarrangement to provide the two modes of operation, e.g. a gas filledcavity which can be selectively filled with an optically diffusing ortransparent gas or a mechanical device which can switch dispersiveelements into and out of the plane of the surface (e.g. in a mannerwhich is analogous to a Venetian blind). In all these examples, thesurface can be electrically switched between two modes, one in which itis substantially diffuse to visible light one and one in which it issubstantially transparent to visible light. The switchable surface may,in some examples, also be able to be switched into intermediate modes inwhich it has differing degrees of diffusivity.

In some examples, the whole of the surface 102 may be switched betweenmodes (in blocks 202 and 204) and in other examples, only part of thescreen may be switched between states. Depending on the granularity ofcontrol of the area which is switched, in some examples, a transparentwindow may be opened up in the surface (e.g. behind an object placed onthe surface) whilst the remainder of the surface stays in itssubstantially diffuse state. Switching of portions of the surface may beuseful where the switching speed of the surface is below the flickerthreshold to enable graphical data (such as an image or graphical userinterface) to be displayed on a portion of the surface whilst projectionoccurs through a different portion of the surface and onto an object.

In other examples, the surface may not be switched between modes but themode of operation may be dependent on the nature of the light incidentupon the surface. For example, the surface may act as a diffuser for oneorientation of polarized light and may be transparent to anotherpolarization. In another example, the optical properties of the surface,and hence the mode of operation, may be dependent on the angle ofincidence of the incident light.

Although FIG. 1 shows a flat surface, in other examples the switchablesurface 102 may be curved or non-planar. The surface may be rigid orflexible. Furthermore, the surface computing device 100 may compriseadditional elements, such as an capture device, not shown in FIG. 1.

As described above, when the system is in projection mode, graphicaldata may be projected through the surface and onto an object 103 (block203). This object may be in contact with the surface, close to thesurface or distant from the surface. The object onto which the graphicaldata is projected may be designed to redirect the light (e.g. throughreflection/refraction) to project the graphical data onto a face of theobject. The face onto which the graphical data is projected may beparallel to the switchable surface or may not be parallel to thesurface. In another example, the projected light may be redirected bythe object such that the light is emitted from the object at an anglewhich is different to the angle at which the light is projected throughthe surface and this may enable projection onto a further object. Theobjects may be passive or may be active (i.e. contain electroniccircuitry) and examples are described below.

The projection through the surface, which may be of different graphicaldata to that projected onto the surface (in display mode) may be used toprovide an alternative display surface (e.g. to provide a privatedisplay), to augment the appearance of the object (e.g. to project animage onto a piece of paper or an animated face onto a games piece), forsensing (e.g. for touch detection on the object or for a beam-breaksensor) or for other purposes. The objects may provide tangible userinterface (UI) controls for the surface computing device or otherwiseprovide a user input to the surface computing device. Examples ofvarious objects and their uses are described below.

FIG. 3 shows schematic diagrams of various passive objects whichredirect the light which enters the object on one face onto another faceof the object. This light has been projected through the switchablesurface in projection mode. A first example of a passive object 301comprises a right-angled isosceles prism and light incident on thebottom face 302 is reflected through 900 as shown in FIG. 3. This relieson total internal reflection (TIR) properties of prisms, although anobject with a mirrored surface may alternatively be used. If thevertical face 303 of the object is frosted or otherwise provided with adiffusing or scattering layer, any graphical data projected onto thebottom face 302 of the prism will be displayed on the vertical face 303.Such an object 301 may therefore be used to provide a private displayfor a user (e.g. to display gaming pieces or confidential information),particularly where the surface computing device is a multi-user device.

A second example of a passive object 311 comprises a narrow section ABCof a prism 308 which has been swept through 3600 about point C. Lightrays incident on the flat bottom face 31 2 (EO, OG) will again undergototal internal reflection (on face DO, OF) and emerge from the outercurved surface 313 (DE, FG). If a two dimensional (2D) image (or othergraphical data) is projected onto the object's bottom flat surface 312,the image will be reflected out through the curved sides of the object;however if suitable diffuser material is attached to the curved surface(or a diffusing surface finish provided), the emerging light will forman image on the curved surface which is visible to the human eye.

FIG. 4 shows schematic diagrams of variants of object 311 where thelight which is incident on the bottom face is redirected onto more thanone face, resulting in graphical data being displayed on more than oneface of the object. A first variant 401 comprises an outer cylindricalsection having parallel top and bottom faces 410, 411. Any lightprojected into this cylindrical region from below will emerge from thetop surface of the object and by placing a diffuser on the top surface(or providing a diffusing surface finish) graphical data may besimultaneously be formed on both the outer curved surface 412 and a ringon the top surface 410. As the two projected elements of graphical data(on surfaces 412, 410) are derived from different areas of the incidentgraphical data (on the bottom face of the object 411, 413), theprojected elements of graphical data may be controlled independently.

A second variant 402 also enables projection onto both the top surface420 and the curved outer surface 421, but this variant enablesprojection onto a central region of the top surface (instead of an outerring as shown in the previous variant 401). The centre section of theobject has a surface 422 which is parallel to the bottom surface 423.The graphical data may be made visible on the top surface through theaddition of a diffusing layer 424 or alternatively by adding a diffusinglayer (or surface) finish to the flat central surface 422. A furthervariant, not shown in FIG. 4, is a combination of the two variants 401,402 described above. This provides three different projection areas: thecurved outer face, an outer ring on the top surface and a centralportion of the top surface, and as described above, projection onto eacharea may be controlled independently.

A further variant 403 is also shown in FIG. 4, which comprises a lens,such as concave lens 430, in the central section of the object.Graphical data projected from below onto this region is expanded to fillthe whole (or a large part) of the top surface 431 of the object and asdescribed above, a diffusing layer 432 may be provided. In such anexample, the size of the projected graphical data is traded off againstthe resolution of the projected graphical data, i.e. the lens increasesthe size of the projected graphical data on the top surface but theprojected resolution remains the same, so the effective resolution ofthe graphical data on the top surface will be less than the resolutionof the graphical data projected onto the curved sides 433. The lens mayintroduce distortion, but this can be corrected for in the graphicaldata that is projected. Whilst a concave lens is shown in FIG. 4, inother examples other lenses may be used, such as a compound lens or aconvex lens.

It will be appreciated that the examples described above are just someof many different example objects which may be used in conjunction witha switchable surface layer to direct the light projected through thesurface and provide projection on different, and in some cases, multiplesurfaces of the object. Dependent on the design of the object and theprojected graphical data, the graphical data may be projected over allthe surface (or surfaces) of the object. Whilst the examples shown havea flat upper surface, this is by way of example only, and other objectsmay comprise a curved upper surface. In a further example, the objectmay be substantially hemispherical.

Although the objects may be in contact with the switchable surface, inother examples, the objects may be separated from the surface e.g. theobjects shown in FIG. 4 may have legs or other members to space themfrom the switchable surface. Where the projection apparatus (which may,for example, comprise more than one projector and switchable filters)and surface can be switched at a rate which exceeds the flickerthreshold, this may enable different graphical data to be projected ontoand under the object.

The through surface projected graphical data may be changed dependent onthe detected position of the object on to which the graphical data isprojected. This detected position may be a position in a planesubstantially parallel to the switchable surface (e.g. an x-y position),the separation (e.g. a z position) between the object and the surfaceand/or the orientation of the object (e.g. the tilt or rotation). In afirst example, a first color may be projected onto a diffuse object whenit is in contact with the surface and a second color may be projected onto the object when it is not in contact with the surface. In a secondexample, a zoom effect may be applied to the projected graphical data asthe object is moved towards or away from the surface, such that theobject is analogous to a magnifying glass. In a third example, differentgraphical data may be projected onto the object dependent upon itsposition (as in the magic lens example described above).

The contact between the diffuse object and the surface may be detectedin any manner (e.g. using a touch detection method). When the diffuseobject is lifted off this surface this may also be detected using touchdetection or alternatively through depth detection (e.g. using a time offlight camera or detection of a structured light pattern projected ontothe object). When a change in the separation of the object and thesurface is detected, the projected light may be changed, for example toa different color or different graphical data. In an example, theprojected graphical data may be adjusted so that it remains in focus orremains the same size on the object etc. This provides 3D basedinteractions and 3D display.

The objects used in combination with the switchable surface may haveintegrated features or embedded electronics to enable their positionrelative to the surface to be tracked. Examples include use of passive(retro-reflective) or active (powered LED) tags, which may use visibleor infra-red (IR) light or light of another wavelength. Other examplesinclude use of wireless communications (e.g.RFID tags). Further examplesare described in more detail below.

In an example shown in FIG. 5, an object 501 may have LEDs 502 or otherlight sources embedded in them (e.g. at the corners) and the position ofthe object may be tracked by detecting the position of the LEDs throughthe switchable surface when in projection mode. In order to achievethis, a camera 503, image capture device or other imaging apparatus maybe located behind the switchable surface 102. Where there is more thanone object in proximity to the surface, different objects may usedifferent flashing patterns (or different modulation schemes) in orderto assist in distinguishing between them. In another examples, the LEDsin an object may be switched on/off in response to a received signal(e.g. a radio signal) and this may be used to distinguish betweenobjects. In a further example, the object may comprise reflectiveelements (e.g. instead of active LEDs) which reflect at least a portionof the light projected through the switchable surface in projectionmode. The light reflected from these elements may be detected by acamera or other imaging apparatus to enable the position of the objectto be tracked.

An object may comprise other active electronics, in addition to orinstead of LEDs or other light sources. For example, an object maycomprise embedded electronics or devices to assist in determining theposition and/or orientation of the object, such as a compass, anaccelerometer, a tilt switch etc, and the object may communicate datafrom these sensors to the surface computing device using a wirelesscommunications technology (e.g. IrDA, Bluetooth™, WiFi etc). Suchdevices may be used instead of, or in addition to, detection of lightemitted or reflected by an object. The location and/or orientation ofthe object may be detected with reference to the switchable surface orwith respect to another object or direction (e.g. with respect togravity). Where light is emitted/reflected at the corners of an object,the additional data may be used to resolve ambiguities which may becaused by incomplete vision data (e.g. the image capture device has onlycaptured three of the four corners) or to provide betterposition/tracking data. In another example, an object may comprise acamera or other image capture device and may communicate data which isindicative of the captured image to the surface computing device. Bycorrelating the received data with any graphical data displayed on theswitchable surface or projected through the surface, the surfacecomputing device may determine the relative position of the object withrespect to the surface.

Where an object is tracked, or its position otherwise determined, thegraphical data projected onto the object may be modified according tothe position of the object. In an example, the graphical data may beadjusted to correct for distortion which may be position dependent (e.g.as described in more detail below) or the projected graphical data maybe different dependent on the position of the object (e.g. to provide anenlarged view of part of the graphical data displayed on the switchablesurface in display mode, to provide a virtual magnifier). This providesa 3D display and enables 3D interactions by a user (e.g. where the useris moving the object relative to the switchable surface).

The objects which are used in combination with the switchable surfacemay enable user input to the surface computing device through detectionof their position on the surface (e.g. using object tracking asdescribed above) and/or through detection of user interaction with theobjects. Two example arrangements 601, 602 which enable detection ofuser interaction with an object are shown in FIG. 6. In the firstexample 601, two prisms 610, 611 placed on the surface 102 may be usedto provide a beam-break sensor. Light projected through the surface inprojection mode and into the base of one of the prisms 610 is reflectedand passes across a gap 612 between the prisms. A camera or otherimaging device may be used to detect light which is reflected backthrough the surface (in projection mode) by the second prism 611. Thebeam-break sensor may use visible light or may use another wavelength oflight, such as infra-red (IR) radiation.

FIG. 6 shows a second arrangement 602 which enables user input throughdetection of a user touching an object 620 which is on (or near) thesurface 102. This touch detection may be achieved by shining light onthe base of the object (e.g. from a source 622 through the switchablesurface 102) which is then redirected by the angled surfaces (asdescribed above). When a user is touching the object, light incident ontheir fingers 621 is reflected and may be detected by an imagingapparatus 623. In an example, IR light may be used, such that the source622 is an IR source and the imaging apparatus is an IR imagingapparatus, such as an IR sensitive camera. The projection and detectionof the light may be performed when the surface 102 is in projection modeor alternatively, where IR light is used, the surface may be at leastpartially transparent to IR light in display mode, thereby allowing theprojection and detection to be performed in display mode. Visible lightmay be projected through the surface in projection mode to projectgraphical data onto the object in addition to using it to provide userinput.

In further examples, active objects (i.e. objects comprising electroniccircuitry) may be used in conjunction with the switchable surface toprovide user input. For example, although the arrangements 601, 602 inFIG. 6 show the light (e.g. IR light) being projected into the objects,in other examples the objects may be active objects and may compriselight sources (e.g. IR LEDs).

In an example, the active object may comprise a touchscreen. Thistouchscreen may use any suitable touchscreen technology, such asresistive or capacitive technology or FTIR (frustrated total internalreflection) as shown in FIG. 7 and described below. In an example, atouchscreen may comprise a switchable surface.

In an example 700, as shown in FIG. 7, an object 701 may comprise alight source 702 which couples light into the object such that itundergoes TIR within the object. When a user touches the object, the TIRis frustrated and light is scattered out of the object 703. Thisscattered light 703 may be detected by an imaging apparatus 704 whichmay be located on the other side of the switchable surface 102. Thedetection of the scattered light may be used to provide user input tothe surface computing device and the user input may also be dependent onthe position of the detected scattered light. The light used may bevisible light or may be IR light. The device may further comprise aprojector 101 arranged to project graphical data onto the object 701 inprojection mode. In a further example 710, the object may be passive andlight used for touch detection based on FTIR may be projected throughthe surface and coupled into the object using a mirrored face orprismatic element. The objects 701, 710 may also comprise regions wherethe surface is roughened (or otherwise treated) in order to frustratethe TIR. Light scattered by such regions may be detected by an imagingapparatus (e.g. apparatus 704) and used to track the position of theobject relative to the switchable surface 102.

The first two arrangements 700, 710 shown in FIG. 7 enable tracking offingers in contact with the front surface of the object (i.e. thesurface which is distant from the switchable surface). A thirdarrangement 720 shown in FIG. 7 enables tracking of fingers on theopposite side of the object (i.e. on the surface of the object closestto the switchable surface, which may be referred to as the back or thebottom surface). The object 721 comprises a layer 722 in which TIRoccurs, where the IR light may be provided by LEDs in the object (as inarrangement 700) or may projected onto and coupled into the object (asin arrangement 710). The object 721 also comprises an IR reflectivesurface 723. When a user touches the back surface 724 of the object(e.g. using a fingertip 725), the TIR is frustrated and the scattered IRlight is reflected by the reflective surface 723 and detected using IRimaging apparatus 704.

In a further example, user inputs may be provided by flexing orotherwise distorting the object onto which light is projected throughthe switchable surface in projection mode. The flexing of the object maybe detected by tracking the position of the object or parts of theobject (e.g. the corners and/or edges) and this may be achieved usingone of the techniques described above. In another example, as shown inFIG. 8, the flexing may be detected through detection of a change inoptical properties of the object, e.g. using polarized opticaldistortion. A polarized light source, which may comprise projector 101(or another light source) and a polarizer 803 may be used to projectpolarized light onto the object 801, which may, for example, be apolycarbonate or acrylic sheet. Having passed through the object, thelight may be reflected by a reflective coating 802 on the front surface(i.e. the surface closest to the user and distant from the switchablesurface 102) and pass back through the polarizer 803 to be detected(e.g. using a camera system 804). Any suitable object may be used wherethe effect on the polarization of light when passing through it isrelated to the amount of stress in the material. When forces are appliedby a user to the object, the optical polarizing properties of the sheetchange such that the amount of rotation of polarization of the lightpassing through any part of the object is dependent on the strain (i.e.different parts of the sheet will experience different strain which willresult in a different change to the polarization of light passingthrough it). As a result, the detected image provides a map of thestrain on the sheet which may be interpreted to determine the useraction (or resultant force) which caused it.

In an example, a touchscreen may be used in combination with the surfacecomputing device to create a document which appears to a user to betouch-sensitive. In such an example, a touchscreen which is transparent(or comprises a switchable surface) may have a printed document placedon top of it. The touchscreen may detect a user touching the documentand the projector in the surface computing device may project additionsonto the document, e.g. in response to the detected user inputs via thetouchscreen.

In the examples described above, the graphical data projected throughthe switchable surface is displayed on the object which is on or nearthe surface and which redirects the light. In other examples, the objectmay redirect the light onto another projection surface, such as a wall,ceiling or projection screen. In some examples, projection screens maybe placed appropriately for use as extended projection space around thesurface computing device, as shown in FIG. 9. In order to be able toproject onto an alternative projection surface 900, one of theprojectors may be orientated such that is off-axis. The system of FIG. 9also comprises a separate projector 101 for projection of graphical dataonto the surface in display mode.

Whilst FIG. 9 shows a prismatic object 902, the object may alternativelycomprise a mirrored surface or may redirect the light in any other way.In an example, light may be projected onto a mirrored surface (e.g.mounted on the ceiling above the switchable surface) and reflected ontoan object (e.g. onto the top surface of an object and/or onto an objectwhich is not optically transparent). The mirrored surface may besteerable and may be steered to track any movement of the object (whichmay be tracked as described above). In a further example, an object maybe used which provides 360° projection (e.g. a hemisphere silvered onthe outside or a cone mirrored on the outside).

In some examples, the position of the object which is used to redirectthe light onto an alternative projection surface may be tracked toenable the projected graphical data to be compensated. By tracking theangle of the object relative to the switchable surface, the projectedgraphical data may be modified such that the projected graphical datamaintains a constant orientation, size or shape (e.g. rectangular ratherthan being a different parallelogram shape or trapezoid). In a variationon such an example, this may be implemented in a virtual manner.Graphical data may be projected onto an object through the surface inprojection mode (as before) and the object may be tracked as it ismanipulated by a user. Another projector may then be used to project thegraphical data directly onto the alternative projection surface (i.e.without being redirected by the object) where the position of thedirectly projected graphical data may be dependent on the position ofthe object relative to the surface computing device. The user perceptionmay be the same for both these examples.

In a further example, the position of users around the switchablesurface may be tracked and the projected graphical data may be modifiedto avoid projecting graphical data onto faces of users. This may beimplemented by adding black regions to the graphical data projectedwhich are positioned and adjusted according to the tracked position of auser's face. This tracking of users may be implemented using imagingapparatus 904.

Some of the objects described above include embedded electronics orother active devices, such as LEDs. The power for these devices may beprovided by a battery located within the object. Alternatively, or inaddition, power may be provided wirelessly by the surface computingdevice. In an example, an object may comprise a photovoltaic cell orother element capable of converting incident optical energy intoelectrical energy. The incident optical energy may be projected throughthe switchable surface in projection mode and the resultant electricalenergy may, for example, be used to power LEDs or other light sources.In an alternative arrangement, power may be inductively coupled into theobject from the surface computing device or other wireless poweringtechniques may be used. In some examples, the power which is provided bythe surface computing device (e.g. inductively or throughoptical-electrical conversion) may be used to recharge a battery withinthe device. In order to reduce the power consumption of an activedevice, the active elements may only be powered for part of the time. Inan example, the active devices may be powered when the switchablesurface is in projection mode.

In some examples, the projected graphical data may be distorted due tothe divergence of the rear-projected graphical data. When an object isplaced on the switchable surface on the optical axis of the projectedgraphical data, the rays of light entering the object are nearperpendicular to the top/bottom faces of the object. However, as shownin a first arrangement 1000 in FIG. 10, when an object 1001 is placed onthe switchable surface 102 off the optical axis 1002, the light enteringthe object 1001 is no longer perpendicular to the bottom face, and asshown in FIG. 10, this may cause the incident light to pass through theobject without being incident on the prism. A solution to this is toinclude a Fresnel lens 1011 that is centered on the optical axis andthat covers substantially the whole of the projection area of theswitchable surface. The Fresnel lens may be selected so that its focallength is equal to the distance between the projector and the switchablesurface. The result is that the divergent light from the projector 101is focused into parallel rays and this emerges perpendicular to thesurface, as shown in the second arrangement 1010 in FIG. 10.

A Fresnel lens 1011, such as that shown in FIG. 10, provides a thin formfactor lens, but in other examples alternative lenses may be used (e.g.a collimating lens or shallow GRIN lens). In a further example 1020, aparabolic mirror 1021 may be used to provide a collimated projectedimage through the switchable surface in projection mode, as shown inFIG. 10 or alternatively an array of individual mirror elements may beused to provide the same optical effect. In the example shown in FIG.10, an imaging apparatus, where required, may be located in front of theparabolic mirror (e.g. at position 1022), behind an aperture in theparabolic mirror 1023, beside the projector 101 directed at theparabolic mirror 1021 or elsewhere. Other optical techniques mayalternatively be used to provide a collimated projected beam.

In an alternative solution to use of a lens, the graphical data may betracked (as described above) and the projected graphical data may beadjusted based on the detected position of the object in order tocorrect for any distortion. In a further alternative solution, theprojector which projects through the surface and onto objects may belocated on a movable mount (e.g. on an x-y stage) and the position ofthe projector may be changed to track the position of the object ontowhich it is projecting. In yet a further example, multiple projectorsmay be provided and a projector may be selected to project the graphicaldata according to a tracked location of the object (e.g. the projectorwith an optical axis which is closest to the position of the object maybe used).

The objects in the examples described above are monolithic objects;however in other examples, an object may comprise a mechanical joint. Inan example, an object may comprise a mouse replacement device andinclude a scroll wheel, the position of which may be tracked (asdescribed above) to provide a user input to a surface computing device.In further examples, the object may be an animate object (e.g. aperson), may not be a solid (e.g. may be a mist) or may be a holographicdevice. The object may be rigid or flexible (e.g. a flexible fiberguide).

The surface computing device may comprise additional elements to thoseshown in the FIGS. and described above. For example, the surfacecomputing device may comprise an image capture device or other imagingapparatus arranged to perform imaging through the surface in projectionmode and/or imaging in display mode (when the surface is substantiallydiffuse). The imaging may use any wavelength, such as visible or IRradiation.

FIG. 11 is a flow diagram showing an example method of operation of asurface computing device, such as any of the devices described hereinand shown in FIGS. 1 and 5-10, which may operate with an object, such asany of the objects described herein and shown in FIGS. 1 and 3-10. Withthe surface in its transparent state (as switched in block 1101), whichis referred to as projection mode, graphical data is projected throughthe surface onto an object (block 1102). The object may be in contactwith the surface, near the surface or distant from the surface. Inaddition to projecting graphical data through the surface in projectionmode, graphical data may be captured through the surface (block 1103).This image capture (in block 1103) may include illumination of thesurface (not shown in FIG. 11). Where devices include electroniccircuitry, the electronic circuitry may be switched on (block 1104) whenthe surface is in transparent state. The captured image (from block1103) may be used in detecting the location of objects through thesurface (block 1105) or alternatively, the location may be determined(in block 1105) based on information received from the electronicdevices within the object. Based on the detected location (from block1105), the projected graphical data may be changed (block 1106). Thecaptured image (from block 1103) and/or the detected location of theobject (from block 1105) may be used to identify a user input (block1109) and this may then be used to control a program (e.g. anapplication) running on the surface computing device (block 1110). Withthe surface in its diffuse state (as switched in block 1107), which isreferred to as display mode, graphical data is projected onto thesurface (block 1108).

The process may be repeated, with the surface (or part thereof) beingswitched between modes (i.e. between diffuse and transparent states) atany rate. In some examples, the surface may be switched at rates whichexceed the threshold for flicker perception. In other examples, whereprojection through the surface only occurs periodically, the surface maybe maintained in display mode (i.e. in its diffuse state) untilprojection is required and then the surface may be switched toprojection mode (i.e. switched to its transparent state).

FIG. 12 illustrates various components of an exemplary surfacecomputing-based device 1200 which may be implemented as any form of acomputing and/or electronic device, and in which embodiments of themethods described herein (e.g. as shown in FIGS. 2 and 11) may beimplemented.

Computing-based device 1200 comprises one or more processors 1201 whichmay be microprocessors, controllers or any other suitable type ofprocessors for processing computing executable instructions to controlthe operation of the device in order to operate as described above (e.g.as shown in FIG. 2 or 11). Platform software comprising an operatingsystem 1202 or any other suitable platform software may be provided atthe computing-based device to enable application software 1203-1208 tobe executed on the device.

The application software may comprise one or more of:

-   -   A display module 1204 arranged to control the projectors 101,        106 (and potentially any shutters 107, 108 associated with the        projectors and FTIR light sources and wireless communications to        objects in the field of view);    -   A surface module 1205 arranged to cause the switchable surface        102 to switch between modes (i.e. between substantially        transparent and diffuse states);    -   An image capture module 1206 arranged to control an image        capture device 1210;    -   An object detection/tracking module 1207 arranged to determine,        and in some cases additionally track, the position of an object        relative to the surface 102; and    -   A touch detection module 1208 arranged to detect touch events        (e.g. as described above with reference to FIG. 6 or 7).        Each module is arranged to cause the switchable surface computer        to operate as described in any one or more of the examples        above.

The computer executable instructions, such as the operating system 1202and application software 1203-1208, may be provided using anycomputer-readable media, such as memory 1209. The memory is of anysuitable type such as random access memory (RAM), a disk storage deviceof any type such as a magnetic or optical storage device, a hard diskdrive, or a CD, DVD or other disc drive. Flash memory, EPROM or EEPROMmay also be used. The memory may also comprise a data store 1211 whichmay be used to store captured images and/or digital data for displayetc.

The computing-based device 1200 also comprises a switchable surface 102,one or more projectors 101, 106 and, in some examples, one or more imagecapture devices 1210. The device may further comprise one or moreadditional projectors, an FTIR subsystem, wireless subsystem (e.g. tocommunicate with the objects), switchable shutters, a light source etc.The computing-based device 1200 may further comprise one or more inputs(e.g. of any suitable type for receiving media content, InternetProtocol (IP) input etc), a communication interface and one or moreoutputs such as an audio output.

Although the present examples are described and illustrated herein asbeing implemented in a surface computing system, the system described isprovided as an example and not a limitation. As those skilled in the artwill appreciate, the present examples are suitable for application in avariety of different types of computing systems.

The above examples refer to use of passive or active objects incombination with a switchable surface. It will be appreciated thatpassive and active objects may combined and an object may have a portionwhich operates passively and a portion which operates actively (due tothe electronic circuitry within the object). Furthermore, objects may bestacked, such that the graphical data projected through a first objectis also projected onto (and in some examples through) a second objectand each of these objects may be passive or active.

The above examples show the switchable surface being orientatedhorizontally and describe the location of objects as being above/belowor in front/behind the surface. It will be appreciated that thisorientation is shown and described by way of example only and thesurface may be positioned in any orientation. Furthermore, as describedabove, the surface may not be planar but may be curved and/or flexible.In an example, the surface computing device may be mounted such that theswitchable surface is vertical and the objects onto which graphical datais projected (in projection mode) may be a user's hands.

The term ‘computer’ is used herein to refer to any device withprocessing capability such that it can execute instructions. Thoseskilled in the art will realize that such processing capabilities areincorporated into many different devices and therefore the term‘computer’ includes PCs, servers, mobile telephones, personal digitalassistants and many other devices.

The methods described herein may be performed by software in machinereadable form on a tangible storage medium. The software can be suitablefor execution on a parallel processor or a serial processor such thatthe method steps may be carried out in any suitable order, orsimultaneously.

This acknowledges that software can be a valuable, separately tradablecommodity. It is intended to encompass software, which runs on orcontrols “dumb” or standard hardware, to carry out the desiredfunctions. It is also intended to encompass software which “describes”or defines the configuration of hardware, such as HDL (hardwaredescription language) software, as is used for designing silicon chips,or for configuring universal programmable chips, to carry out desiredfunctions.

Those skilled in the art will realize that storage devices utilized tostore program instructions can be distributed across a network. Forexample, a remote computer may store an example of the process describedas software. A local or terminal computer may access the remote computerand download a part or all of the software to run the program.Alternatively, the local computer may download pieces of the software asneeded, or execute some software instructions at the local terminal andsome at the remote computer (or computer network). Those skilled in theart will also realize that by utilizing conventional techniques known tothose skilled in the art that all, or a portion of the softwareinstructions may be carried out by a dedicated circuit, such as a DSP,programmable logic array, or the like.

Any range or device value given herein may be extended or alteredwithout losing the effect sought, as will be apparent to the skilledperson.

It will be understood that the benefits and advantages described abovemay relate to one embodiment or may relate to several embodiments. Theembodiments are not limited to those that solve any or all of the statedproblems or those that have any or all of the stated benefits andadvantages. It will further be understood that reference to ‘an’ itemrefers to one or more of those items.

The steps of the methods described herein may be carried out in anysuitable order, or simultaneously where appropriate. Additionally,individual blocks may be deleted from any of the methods withoutdeparting from the spirit and scope of the subject matter describedherein. Aspects of any of the examples described above may be combinedwith aspects of any of the other examples described to form furtherexamples without losing the effect sought.

The term ‘comprising’ is used herein to mean including the method blocksor elements identified, but that such blocks or elements do not comprisean exclusive list and a method or apparatus may contain additionalblocks or elements.

It will be understood that the above description of a preferredembodiment is given by way of example only and that variousmodifications may be made by those skilled in the art. The abovespecification, examples and data provide a complete description of thestructure and use of exemplary embodiments of the invention. Althoughvarious embodiments of the invention have been described above with acertain degree of particularity, or with reference to one or moreindividual embodiments, those skilled in the art could make numerousalterations to the disclosed embodiments without departing from thespirit or scope of this invention.

1. A surface computing device comprising: a surface layer having atleast two modes of operation, wherein in a first mode of operation thesurface layer is substantially diffusing to visible light and in asecond mode of operation, the surface layer is substantially transparentto visible light; and projection apparatus operable to project graphicaldata onto the surface layer in the first mode of operation and toproject graphical data through the surface onto an object in the secondmode of operation.
 2. A surface computing device according to claim 1,wherein the projection apparatus is arranged to project differentgraphical data in each mode of operation.
 3. A surface computing deviceaccording to claim 1, further comprising: object detection apparatusarranged to detect at least one of a location and an orientation of theobject relative to the surface layer.
 4. A surface computing deviceaccording to claim 3, wherein the graphical data projected through thesurface onto an object in the second mode of operation is dependent uponthe detected at least one of a location and an orientation of theobject.
 5. A surface computing device according to claim 1, furthercomprising: imaging apparatus arranged to capture an image through thesurface layer in the second mode of operation.
 6. A surface computingdevice according to claim 5, further comprising: a processor; and memoryarranged to store executable instructions to cause the processor to:process said image to identify a user input; and provide an input to aprogram running on the surface computing device according to said userinput.
 7. A surface computing device according to claim 1, wherein theprojection apparatus comprises a display device operable to projectgraphical data onto the surface layer in the first mode of operation anda projector operable to project graphical data through the surface ontoan object in the second mode of operation.
 8. A method of operating asurface computing device comprising: switching a surface layer between asubstantially diffuse state and a substantially transparent state; inthe substantially diffuse state, projecting graphical data onto thesurface layer; and in the substantially transparent state, projectinggraphical data through the surface onto an object.
 9. A method accordingto claim 8, further comprising: detecting a position of the objectrelative to the surface layer.
 10. A method according to claim 9,further comprising: changing the graphical data projected through thesurface onto the object according to the detected position.
 11. A methodaccording to claim 8, further comprising: in the substantiallytransparent state, capturing an image through the surface.
 12. A methodaccording to claim 11, further comprising: processing said image toidentify a user input; and controlling a program running on the surfacecomputing device according to said user input.
 13. A surface computingsystem comprising: a layer that can be electrically switched between afirst state which is substantially diffusing to visible light and asecond state which is substantially transparent to visible light; andprojection apparatus, comprising at least one projector, arranged toproject graphical data onto the layer in its first state and to projectgraphical data through the layer onto an object in its second state. 14.A surface computing system according to claim 13, further comprising theobject and wherein the object is operable to redirect the graphical dataprojected onto the object.
 15. A surface computing system according toclaim 14, wherein the object comprises a display surface and wherein theobject is operable to redirect the graphical data onto the displaysurface.
 16. A surface computing system according to claim 15, whereinthe object comprises a plurality of display surfaces and wherein saidgraphical data projected through the layer in its second state comprisesgraphical data for display on at least one of the plurality of displaysurfaces.
 17. A surface computing system according to claim 13, furthercomprising: object detection apparatus arranged to detect at least oneof a position and an orientation of the object.
 18. A surface computingsystem according to claim 17, wherein the projection apparatus isfurther arranged to project different graphical data onto an objectdependent upon the detected at least one of a position and anorientation of the object.
 19. A surface computing system according toclaim 17, wherein the object detection apparatus comprises an imagecapture device.
 20. A surface computing system according to claim 13,further comprising: touch detection apparatus arranged to detect a usertouching the object and to control a program running on the surfacecomputing system in response to said detection of a user touching theobject.